A Communication Efficient and Scalable Distributed Data Mining for the Astronomical Data

摘要

In 2020, ~60PB of archived data will be accessible to the astronomers. But toanalyze such a paramount data will be a challenging task. This is basically dueto the computational model used to download the data from complexgeographically distributed archives to a central site and then analyzing it inthe local systems. Because the data has to be downloaded to the central site,the network BW limitation will be a hindrance for the scientific discoveries.Also analyzing this PB-scale on local machines in a centralized manner ischallenging. In this virtual observatory is a step towards this problem,however, it does not provide the data mining model. Adding the distributed datamining layer to the VO can be the solution in which the knowledge can bedownloaded by the astronomers instead the raw data and thereafter astronomerscan either reconstruct the data back from the downloaded knowledge or use theknowledge directly for further analysis.Therefore, in this paper, we presentDistributed Load Balancing Principal Component Analysis for optimallydistributing the computation among the available nodes to minimize thetransmission cost and downloading cost for the end user. The experimentalanalysis is done with Fundamental Plane(FP) data, Gadotti data and complexMfeat data. In terms of transmission cost, our approach performs better thanQi. et al. and Yue.et al. The analysis shows that with the complex Mfeat data~90% downloading cost can be reduced for the end user with the negligible lossin accuracy.